Skip to content

Refresh meos-idl.json to MEOS-API canonical; regenerate bindings#2

Merged
Nyuke235 merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/meos-idl-refresh-extended-types
May 27, 2026
Merged

Refresh meos-idl.json to MEOS-API canonical; regenerate bindings#2
Nyuke235 merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:feat/meos-idl-refresh-extended-types

Conversation

@estebanzimanyi
Copy link
Copy Markdown
Member

@estebanzimanyi estebanzimanyi commented May 22, 2026

Refreshes codegen/res/meos-idl.json to the MEOS-API canonical IDL and regenerates both outputs, now tracking the public MobilityDB surface.

Source of truth (reproducible)

  • MEOS-API: master, including #15 (PG-vendored type recovery)
  • MobilityDB: master at 2c4243a

The committed IDL is exactly what python setup.py && python run.py produces against those two states:

# fresh MEOS-API master checkout (post-#15)
python setup.py            # clones MobilityDB master
python run.py              # -> output/meos-idl.json : 2672 functions, 47 structs, 6 enums
diff <(jq -S . output/meos-idl.json) <(jq -S . codegen/res/meos-idl.json)   # empty

What changed vs the previous IDL

  1. ~512 signature drifts fixed — regenerating with #15's typerecover.py recovers GSERIALIZED / Datum / DateADT / GBOX / BOX3D / AFFINE slots that the earlier parse had collapsed to int / int *.
  2. 2 functions recoveredmindistance_tgeo_tgeo and tgeoarr_tgeoarr_mindist are present.
  3. Tracks public master (2672 functions). The not-yet-merged extended-type families (tbigint / th3index / extended tcbuffer / tpose) are dropped; they re-enter via a follow-up refresh once they land upstream, so the canonical binding never points at non-public code.

Datum stays out of the user API

The recovered types exposed that 171 functions carry a Datum in their signature — all internal (meos_internal.h). A Datum is an opaque tagged uintptr_t (a pointer for Float8 / text / geometry base types, 32-bit under WASM32) and cannot be marshalled to a JS scalar. The generator now skips every Datum-signature function (isDatum added to shouldSkip); the user-facing path is the typed *_meos.c wrappers, e.g. temporal_start_valuetint_start_value / tfloat_start_value / ttext_start_value. No Datum appears in any functions.generated.ts signature.

Wrapper + build sync

  • core/types/basic/{TFloat,TInt,TNumber}.ts: track MobilityDB's mult_*mul_* multiplication rename (the only core/types change — the symbols were renamed upstream).
  • Dockerfile: bump MOBILITYDB_COMMIT to 2c4243a so the WASM build matches the regenerated bindings.c.

Verify

npm run generate    # Generated: 2465  Skipped: 172  Manual: 35
npx tsc --noEmit    # clean

@Nyuke235
Copy link
Copy Markdown
Collaborator

Thank you for the refresh. I ran a final reproducibility check after MobilityDB/MEOS-API#15 landed, and the committed IDL doesn't match what regenerates from the current canonical state. Could you regenerate against master once more before we merge?

Pulled MEOS-API master (now with #15 merged), ran python run.py against a fresh setup.py clone of MobilityDB master, and diffed output/meos-idl.json against codegen/res/meos-idl.json from this branch:

Esteban's IDL: 2830 functions, 47 structs, 6 enums
Fresh IDL:     2672 functions, 47 structs, 6 enums

In Esteban but not Fresh: 160
In Fresh but not Esteban: 2
Common: 2670
Signature drift in common funcs: 512

tsc --noEmit still passes on this branch (I re-confirmed), because the drifts are in functions no core/types/**/*.ts wrapper currently consumes, but the IDL is the binding-generation source of truth, so the next wrapper that touches one of these will get an incorrect signature.

3 causes:

1. Older typerecover.py (the largest of the three)

Timestamps:

  • 23722ba (this PR's refresh commit): 2026-05-22T21:30:29Z
  • 9ef867a (the typerecover.py commit that landed in MEOS-API#15): 2026-05-24T19:18:42Z

The refresh was generated ~46h before _TYPE_MAP reached its final form. The merged version recovers GSERIALIZED, Datum, DateADT, GBOX, BOX3D, AFFINE; the IDL in this PR was generated before those entries existed. Concrete examples:

Function (slot) This PR's IDL Fresh from master
edwithin_tcbuffer_geo (gs param) const int * const GSERIALIZED *
geom_perimeter (gs param) const int * const GSERIALIZED *
temporal_start_value (return) int Datum
tsequence_max_val (return) int Datum
overbefore_set_date (d param) int DateADT

512 functions are affected: every GSERIALIZED * / Datum / DateADT / GBOX * / BOX3D * slot in scope is still spelled int/int * here.

2. MobilityDB headers were not master

The IDL contains 160 functions that don't exist in MobilityDB master:

  • 60 tbigint_*
  • 63 th3index_*
  • 121 tcbuffer_* (vs ~70 on master)
  • 54 tpose_*

These look like a feature branch with the extended type families. Could you confirm which MobilityDB commit/branch was used? If it's a not-yet-merged feature branch, the binding generator's source of truth would point at code that doesn't exist upstream, I'd prefer the canonical MEOS.js to track a state of MobilityDB that's already public.

3. Two functions present on master but missing here

mindistance_tgeo_tgeo and tgeoarr_tgeoarr_mindist — confirms the headers snapshot used was slightly behind current master.

What I'm asking for

  1. Rebase on current main (now with Initial commit #1 merged) and regenerate codegen/res/meos-idl.json via python run.py against:
    • MEOS-API at current master (after the #15 merge)
    • MobilityDB headers at a stable state: either master, or a specific feature branch you can document in the PR body
  2. Re-run npm run generate against the new IDL, commit the updated bindings.c + functions.generated.ts
  3. Force-push the amended 23722ba

The reproducibility check I ran is:

# in a fresh MEOS-API checkout, master at or after the #15 merge
python setup.py                         # or --branch <feature> if you used a branch
python run.py
diff <(jq -S . output/meos-idl.json) \
     <(jq -S . path/to/MEOS.js/codegen/res/meos-idl.json)

Once that diff is empty, I'll re-verify and approve.

@estebanzimanyi estebanzimanyi force-pushed the feat/meos-idl-refresh-extended-types branch from 23722ba to bbf7831 Compare May 26, 2026 21:00
@estebanzimanyi
Copy link
Copy Markdown
Member Author

Done — rebased onto main (now with #1) and force-pushed bbf7831. Regenerated against MEOS-API master (post-#15) and MobilityDB master 2c4243a, and your reproducibility diff is now empty:

$ diff <(jq -S . output/meos-idl.json) <(jq -S . codegen/res/meos-idl.json)
$       # (empty)
Done: 2672 functions, 47 structs, 6 enums

Point by point:

  1. typerecover (512 drifts) — fixed at the root by regenerating with #15. The GSERIALIZED / Datum / DateADT / GBOX / BOX3D / AFFINE slots are recovered; your cited examples now read const GSERIALIZED *, Datum, DateADT.

  2. Source state — I took your preferred option and pointed the regen at public master, so the 160 extended-type functions (tbigint / th3index / extended tcbuffer / tpose) are gone. They sit on a not-yet-merged feature branch; I'd rather refresh them in once they land upstream than have the canonical binding track non-public code.

  3. Missing functionsmindistance_tgeo_tgeo and tgeoarr_tgeoarr_mindist are present.

One thing your Datum examples surfaced: temporal_start_value (and 170 others) now correctly type as Datum — but Datum is internal (meos_internal.h) and can't be marshalled to a JS scalar (it's a pointer for Float8/text/geometry, and only 32-bit under WASM32). So the generator now skips every Datum-signature function, and the user-facing path stays the typed wrappers (tint_start_value / tfloat_start_value / …). functions.generated.ts carries no Datum in any signature.

I also bumped the Dockerfile MOBILITYDB_COMMIT pin to 2c4243a and tracked the upstream mult_*mul_* rename in the TFloat/TInt/TNumber wrappers (the only core/types change).

npm run generate → 2465 generated / 172 skipped / 35 manual; npx tsc --noEmit clean.

Regenerate codegen/res/meos-idl.json from MEOS-API master (including #15's
typerecover fix) against MobilityDB master (2c4243a). This recovers the
PG-vendored C types the libclang parse had collapsed to int — GSERIALIZED,
Datum, DateADT, GBOX, BOX3D, AFFINE — fixing ~512 signature drifts, and picks
up mindistance_tgeo_tgeo / tgeoarr_tgeoarr_mindist. The IDL now tracks the
public MobilityDB surface (2672 functions); the not-yet-merged extended-type
families are dropped until they land on master.

Generate the binding for the PUBLIC MEOS API only. The generator now excludes:
  - internal headers (meos_internal.h / meos_internal_geo.h) — not user-facing,
    unused by core/types, and many carry the opaque Datum;
  - the bundled liblwgeom glue header (postgis_ext_defs.in.h) — PostGIS internals;
  - any Datum-signature function — a Datum is an opaque tagged uintptr_t (a
    pointer for Float8/text/geometry, 32-bit under WASM32) that cannot be
    marshalled to a JS scalar; the user-facing path is the typed *_meos.c wrapper
    (e.g. temporal_start_value -> tint/tfloat/ttext_start_value).
No Datum appears in any functions.generated.ts signature. The C preamble now
includes the public extended-type headers (meos_cbuffer/npoint/pose/rgeo.h) so
the cbuffer/npoint/pose/trgeometry wrappers compile.

Track MobilityDB's mult -> mul multiplication rename in the TFloat/TInt/TNumber
wrappers. Update the Dockerfile WASM build: bump the MOBILITYDB_COMMIT pin to
2c4243a, enable the CBUFFER/NPOINT/POSE/RGEO MEOS modules, and add the json-c
and gsl include paths so the regenerated extended-type bindings compile and link.

  npm run generate -> Generated: 2124  Skipped: 1  Manual: 35
  npx tsc --noEmit  -> clean
  reproducibility: diff of a fresh run.py output vs the committed IDL is empty
@estebanzimanyi estebanzimanyi force-pushed the feat/meos-idl-refresh-extended-types branch from bbf7831 to 395ffa5 Compare May 26, 2026 22:52
@estebanzimanyi
Copy link
Copy Markdown
Member Author

Pushed a refinement to 395ffa5 (force-push) — heads-up on what changed since the last review:

The generated binding now mirrors the PUBLIC MEOS API only. The generator excludes, in addition to Datum-signature functions:

  • the internal headers (meos_internal.h / meos_internal_geo.h) — not user-facing, unused by core/types;
  • the bundled liblwgeom glue header (postgis_ext_defs.in.h) — PostGIS internals.

The C preamble now also #includes the public extended-type headers (meos_cbuffer/npoint/pose/rgeo.h) so those wrappers compile. npm run generate → 2124 generated / 35 manual; tsc --noEmit clean. The committed meos-idl.json is unchanged, so the reproducibility diff against MobilityDB master is still empty — this refinement is generator-side only.

On the WASM build: I verified make release end-to-end — meos.js + meos.wasm build and link cleanly with the extended-type modules (the Dockerfile now enables CBUFFER/NPOINT/POSE/RGEO and adds the json-c/gsl include paths). It does require one upstream fix: five functions are declared in meos_rgeo.h on master but never implemented (trgeo_points, trgeo_rotation, trgeo_segments, trgeo_traversed_area, nad_stbox_trgeo), which I've fixed in MobilityDB/MobilityDB#1124. Once that lands on master, the pinned-master WASM build links with no further changes here.

@Nyuke235
Copy link
Copy Markdown
Collaborator

Re-verified after the rebase and the v3 force-push (395ffa5). Everything checks out, approving.

Reproducibility check (the key one this round), synced my MobilityDB clone to your pin 2c4243a (after the mult_* -> mul_* rename in 4b70d8a), regenerated, and:

Bit-for-bit match. The first round's drift was entirely explained by your IDL being generated against a typerecover state that hadn't yet landed in #15 plus an unpinned MobilityDB; this round, with everything pinned and aligned, the chain is fully reproducible from canonical inputs.

Other checks:

  • npx tsc --noEmit -> 0 errors
  • No Datum in any TypeScript signature, only in comments of the manual overrides (tstzspan_lower, span_eq, adjacent_span_span, tstzspanset_lower, the // SKIP trgeo_restrict_value line). The "no Datum in any signature" claim holds.
  • 2150 exported wrappers in functions.generated.ts, consistent with 2124 generated + 35 manual

LGTM, merging.

@Nyuke235 Nyuke235 merged commit 4217c42 into MobilityDB:main May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants